Control of virtual object using device touch interface functionality

ABSTRACT

A virtual object can be controlled using one or more touch interfaces. A location for a first touch input can be determined on a first touch interface. A location for a second touch input can be determined on a second touch interface. A three-dimensional segment can be generated using the location of the first touch input, the location of the second touch input, and a pre-determined spatial relationship between the first touch interface and the second touch interface. The virtual object can be manipulated using the three-dimensional segment as a control input.

CLAIM OF PRIORITY

This application is a continuation of U.S. patent application Ser. No.14/809,975, filed Jul. 27, 2015, the entire contents of which areincorporated herein by reference.

U.S. patent application Ser. No. 14/809,975 is a continuation of U.S.patent application Ser. No. 12/917,362, filed Nov. 1, 2010 (U.S. Pat.No. 9,092,135, Issued Jul. 28, 2015), the entire contents of which areincorporated herein by reference.

FIELD OF THE INVENTION

Embodiments of the present invention are related to devices that utilizetouch interfaces and methods for controlling virtual objects using thosetouch interfaces.

BACKGROUND OF THE INVENTION

Handheld consumer electronic devices such as smart phones, portableinternet devices, portable music players, and hand held gaming devicesoften include some form of visual display, such as a flat screen videodisplay or a touch screen display. Touch screens are displays which alsohave the ability to detect the location of touches within the displayarea. This allows the display to be used as an input device, removingthe keyboard and/or the mouse as the primary input device forinteracting with the display's content. Such displays can be attached tocomputers or, as terminals, to networks. Touch screens also haveassisted in recent changes in the design of personal digital assistants(PDAs), satellite navigation and mobile phone devices, making thesedevices more user-friendly.

Touch screens have become commonplace since the invention of theelectronic touch interface in 1971 by Dr. Samuel C. Hurst. They havebecome familiar on PDAs, smart phones, and portable internet deviceswhere a stylus or fingers are used to manipulate a graphical userinterface (GUI) and to enter data. The popularity of smart phones, PDAs,portable game consoles, and many types of information appliances isdriving the demand for, and the acceptance of, touch screens.

Currently, most devices that incorporate the use of touch interfaces useonly a single touch interface per device. Because only a single touchinterface is used, the manipulation of virtual objects displayed onthose devices is limited. With a single touch interface, only2-dimensional manipulation of virtual objects may be performedintuitively.

It is within this context that embodiments of the present inventionarise.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1A is a 3-dimensional schematic diagram illustrating a device thatuses direct touch for controlling a virtual object according to anembodiment of the present invention.

FIG. 1B is a top-view illustrating a device that uses direct touch forcontrolling a virtual object according to an embodiment of the presentinvention.

FIG. 1C is a bottom-view illustrating a device that uses direct touchfor controlling a virtual object according to an embodiment of thepresent invention.

FIG. 2A is a 3-dimensional schematic diagram illustrating a device thatuses pre-touch sensors for controlling a virtual object according to anembodiment of the present invention.

FIG. 2B is a top-view illustrating a device that uses pre-touch sensorsfor controlling a virtual object according to an embodiment of thepresent invention.

FIG. 2C is a bottom-view illustrating a device that uses pre-touchsensors for controlling a virtual object according to an embodiment ofthe present invention.

FIG. 3A is a 3-dimensional schematic diagram illustrating a method forcontrolling a virtual object using direct touch according to anembodiment of the present invention.

FIG. 3B is a 3-dimensional schematic diagram illustrating a method forcontrolling a virtual object using pre-touch according to an embodimentof the present invention.

FIG. 4A is a schematic diagram illustrating a method for rotating avirtual object according to an embodiment of the present invention.

FIG. 4B is a schematic diagram illustrating a method for shifting avirtual object according to an embodiment of the present invention.

FIG. 4C is a schematic diagram illustrating a method for squeezing orcompressing a virtual object according to an embodiment of the presentinvention.

FIG. 4D is a schematic diagram illustrating a method for expanding avirtual object according to an embodiment of the present invention.

FIG. 5 is a block diagram illustrating an apparatus for controlling avirtual object according to an embodiment of the present invention.

FIG. 6 illustrates an example of a non-transitory computer-readablestorage medium with instructions for implementing control of a virtualobject according to an embodiment of the present invention.

DESCRIPTION OF THE SPECIFIC EMBODIMENTS

Although the following detailed description contains many specificdetails for the purposes of illustration, anyone of ordinary skill inthe art will appreciate that many variations and alterations to thefollowing details are within the scope of the invention. Accordingly,the exemplary embodiments of the invention described below are set forthwithout any loss of generality to, and without imposing limitationsupon, the claimed invention.

FIG. 1A is a 3-dimensional schematic diagram illustrating a device thatutilizes direct touch for controlling a virtual object in accordancewith an embodiment of the present invention. In this embodiment, a firsttouch interface 103 and a second touch interface 105 are both situatedon a single device 101. In FIG. 1A, the touch interfaces 103, 105 areimplemented as flat surfaces for the purpose of example, but notlimitation. It is important to note that touch interfaces may beimplemented in a variety of shapes and sizes including curved surfaces.The first touch interface 103 is situated on a front of the device 101,and will be referred to herein as the front touch interface. The secondtouch interface 105 is situated on a back of the device 101, and will bereferred to herein as the rear touch interface. In some embodiments thefront touch interface 103 or rear touch interface 105 may be implementedas a touch screen. A touch screen refers to a visual display which hasthe ability to detect the location of contact within the display area.Alternatively, the front touch interface 103 or rear touch interface 105may be implemented as a touch pad. A touch pad refers to a touchinterface that does not include a visual display. Instead, a user'scontact with the touch pad can be communicated via a processor and,optionally, to an independent visual display.

Both the front touch interface 103 and the rear touch interface 105 areconfigured to determine the location of a touch input. By way ofexample, and not by way of limitation, the touch interfaces may beconfigured to determine the location of a touch input using resistivetouch panels, surface acoustic wave technology, capacitive touch panels,infrared touch panels, strain gauge technology, optical imagining,dispersive signal technology, acoustic pulse recognition, or frustratedtotal internal reflection.

A resistive touch panel may be composed of several layers which mayinclude two thin metallic electrically conductive and resistive layersseparated by thin space. When an object (e.g., finger, stylus) comes incontact with this kind of touch panel, the layers are connected at acertain point. The panel then electrically acts similar to two voltagedividers with connected outputs. This causes a change in the electricalcurrent which is registered as a touch event which may be sent to aprocessor for processing.

Surface Acoustic Wave technology uses ultrasonic waves that pass overthe touch screen panel. When the panel is touched, a portion of the waveis absorbed. A sensor registers the resulting change in the ultrasonicwaves and sends corresponding information to a processor for processing,which can determine the position of the touch event.

A capacitive touch panel may be coated with a material, e.g., indium tinoxide that conducts a continuous electrical current across the sensor.The sensor therefore exhibits a precisely controlled field of storedelectrons in both the horizontal and vertical axes. When the capacitivetouch panel's ‘normal’ capacitance field (its reference state) isaltered by an externally applied electric field, e.g., from a user'sfinger, electronic circuits located at each corner of the panel measurea resultant ‘distortion’ in the characteristics of the reference fieldand send the information about the touch event to a processor formathematical processing to determine the position of the touch event.

An infrared touch panel may employ one of two different methodologies.One method uses thermally induced changes of the surface resistance.Another method incorporates an array of vertical and horizontal IRsensors that detect interruption of a modulated light beam near thesurface of the panel.

Some touch interfaces incorporate a spring gauge configuration. In thisconfiguration the touch surface, e.g., a screen or touch pad, is springmounted at each of its four corners and strain gauges are used todetermine the deflection of the screen when touched. The position of thetouch can be calculated from the deflection as determined by the straingauges.

In touch interface technology based on optical imaging, two or moreimage sensors may be placed around the edges (mostly the corners) of thescreen. Infrared backlights may be placed in a camera's field of view onthe other sides of the screen. A touch shows up as a shadow andlocations of the shadow as determined by each pair of cameras can thenbe triangulated to locate the position of the touch.

Dispersive signal technology may use sensors to detect mechanical energyin the glass touch panels that occurs due to a touch. Complex algorithmsare then used to interpret this information to ultimately provide theactual location of the touch.

Touch panels based on acoustic pulse recognition may use more than twopiezoelectric transducers located at some positions on the panel to turnthe mechanical energy of a touch (vibration) into an electronic signal.This signal may then be converted into an audio file, and then comparedto a pre-existing audio profile for every position on the screen inorder to determine the location of the touch.

Touch panels based on frustrated total internal reflection use theprinciple of total internal reflection to fill a refractive medium withlight. When a finger or other soft object is pressed against the surfaceof the panel, the internal reflection light path is interrupted, makingthe light reflect outside of the medium and thus visible to a camerabehind the medium. The camera then processes this information todetermine the location of the touch.

When a first finger F₁ comes in contact with the front touch interface103, the front touch interface 103 will determine the location of thisfirst touch input 107 with respect to the front touch interface 103.Similarly, when a second finger F₂ comes in contact with the rear touchinterface 105, the rear touch interface 105 will determine the locationof this second touch input 109 with respect to the rear touch interface105. By way of example, and not by way of limitation, the touch inputlocations 107, 109 may be determined by selecting the center-mostposition of the entire surface that either finger F₁, F₂ comes incontact with using any of the touch panel technologies described above.It is important to note that a touch input may be realized by contactfrom objects other than fingers (e.g., stylus, controller, etc.)

Once the front touch interface 103 and the rear touch interface 105 havedetermined their respective touch input locations 107, 109, the device101 can then generate a 3-dimensional segment 111 using the location ofthe first touch input 107, the location of the second touch input 109,and a pre-determined spatial relationship between the front touchinterface 103 and the rear touch interface 105. By way of example, andnot by way of limitation, the pre-determined spatial relationshipbetween the front touch interface 103 and the rear touch interface 105may be a multiple of the actual physical distance d between the twointerfaces. In a relatively simple case, the front and rear touchinterfaces may be of substantially the same size and shape and may beassumed to be in a parallel and overlapping relationship with respect toeach other. It is noted that this case is a non-limiting example.

As an example of generation of the three dimensional segment 111consider a case where the touch interfaces 103, 105 are in asubstantially overlapping arrangement separated by a distance d.Locations of the touch inputs may be defined with respect to a commonorigin of an X-Y plane of a Cartesian co-ordinate system, which, for thesake of example, may be located on the surface of the first touchinterface 103. The front touch input location 107. The first touch inputlocation may be defined by x, y, z, coordinates x₁, y₁, and 0. Thesecond touch input location 109 may be defined by x, y, z, coordinatesx₂, y₂, −d. The direction and orientation of the segment 111 may beexpressed as a vector (x₁−x₂)i+(y₁−y₂)j−dk, where i, j, and k are unitsegments in the x, y, and z directions respectively. The orientation ofthe segment 111 may alternatively be expressed in terms of Eulerianangles, i.e., angles of projections of the segment 111 onto the x-y,x-z, and y-z planes relative to the x, y, and z axes respectively. Themagnitude of the segment 111 may be expressed in terms of a length L ofthe segment 111, which may be calculated using the well-known distanceformula, e.g., as L=√{square root over ((x₁−x₂)²+(y₁−y₂)²+d²)}.

The three-dimensional segment 111 is then used as a control input forcontrolling the virtual object 113. By way of example, and not by way oflimitation, the virtual object 113 may appear as though it is locatedbetween the front touch interface 103 and the rear touch interface 105.The virtual object 113 could also appear as though it is located behindthe rear touch interface 105 or could assume any number of differentorientations. For purposes of our example, we will assume that thevirtual object 113 appears to be located between the front touchinterface 103 and the rear touch interface 105. The three-dimensionalsegment 111 will pass through the virtual object 113, and any shift inposition of the three-dimensional motion segment 111 will manipulate theorientation of the virtual object 113. Thus, any shift in the firsttouch input location 107 or any shift in the second touch input location109 will cause the virtual object 113 to transform its orientation inaccordance with the newly created three-dimensional segment 111generated by those shifts.

FIG. 1B is a top-view of the device 101 that utilizes direct touch forcontrolling a virtual object. The first finger F₁ comes in contact withthe front touch interface 103, generating a first touch input location107. That first touch input location 107 will then be used to generate athree-dimensional segment in accordance with the method described above.

FIG. 1C is a bottom-view of the device 101 that utilizes direct touchfor controlling a virtual object. The second finger F₂ comes in contactwith the rear touch interface 105, generating a second touch inputlocation 109. That second touch input location 107 will then be used togenerate a three-dimensional segment in accordance with the methoddescribed above.

FIG. 2A is a three-dimensional schematic diagram of a device that usespre-touch sensors for controlling a virtual object in accordance withanother embodiment of the present invention. Again, the first touchinterface 203 and the second touch interface 205 are situated on thesame device 201. We will refer to the first touch interface 203 as thefront touch interface and the second touch interface 205 as the reartouch interface. Much like the device in FIG. 1A, the front touchinterface 203 and rear touch interface 205 of the present device 201 maybe implemented as either a touch screen or a touch pad.

However, whereas the touch interfaces of FIG. 1A required direct contactto determine the location of touch inputs, the touch interfaces 203, 205of FIG. 2A may generate touch input locations without direct contact.The front touch interface 203 and the rear touch interface 205 both usepre-touch sensors to determine touch input location. These pre-touchsensors may determine a touch input location whenever an object comes inclose proximity to the touch interface, without actually coming incontact with it. For example, when a first finger F₁ comes in proximityto the front touch interface 203, the front touch interface 203 willdetermine the location of this first touch input 207. The location ofthe first finger F₁ in proximity to the front touch interface 203 willbe referred to herein as the first point of application 204. Thelocation of the first touch input 207 is characterized by the respectiveposition of the first point of application on the front touch interfaceas established by the pre-touch sensors, as well as the perpendiculardistance D₁ between the first point of application 204 and itsrespective position on the front touch interface 203. Likewise, when asecond finger F₂ comes in proximity with the rear touch interface 205,the rear touch interface 205 will determine the location of this secondtouch input 209 in a manner equivalent to that used to determine thefirst touch input location.

Pre-touch sensors may be implemented in a variety of ways. By way ofexample, and not by way of limitation, pre-touch sensors may beimplemented through the use of capacitive touch panels, infrared touchpanels, optical imaging, or even surface acoustic wave technology. Theimplementation of these pre-touch technologies was described above. Itis important to note that any touch interface's ability to determineobject location is limited to objects that are within close proximity tothe touch interface.

Once the location of the touch inputs 207, 209 are determined, thedevice 201 will control the virtual object 213 in a manner similar tothat described above. The device 201 generates a three-dimensionalsegment 211 using the location of the first touch input 207, thelocation of the second touch input 209, and a pre-determined spatialrelationship between the front touch interface 203 and the rear touchinterface 205. Alternatively, the three-dimensional segment 211′ may bedetermined between the points of application 204, 206. Thisthree-dimensional segment 211′ may be modified by lengthening orshortening the perpendicular distances D₁, D₂ between the points ofapplication 204, 206 and their respective positions 207, 209 on thetouch interfaces 203, 205. Finally, any shift in position of thethree-dimensional motion segment 211, 211′ will manipulate theorientation of the virtual object 213. It is important to note that thethree-dimensional segment 211′ generated by the points of application204, 206 will provide for a more intuitive and effective user-experiencewhen compared to the three-dimensional segment 211 generated by the twotouch inputs 209, 211. This is because the three-dimensional segment211′ between the points of application 204, 206 possesses an extradegree of control (i.e., perpendicular distance between points ofapplication and respective positions on touch interface)

FIG. 2B is a top-view of the device 201 that uses pre-touch sensors forcontrolling a virtual object. The first finger F₁ comes in proximity tothe front touch interface 203. The location of the first touch input 207is then characterized by the respective position of the first point ofapplication on the front touch interface as established by the pre-touchsensors, as well as the perpendicular distance D₁ between the firstpoint of application 204 and its respective position on the front touchinterface 203. That first touch input location 207 will then be used togenerate a three-dimensional segment in accordance with the methoddescribed above.

FIG. 2C is a bottom-view of the device 201 that uses pre-touch sensorsfor controlling a virtual object. The second finger F₂ comes inproximity to the rear touch interface 205. The location of the secondtouch input 209 is then characterized by the respective position of thesecond point of application on the rear touch interface as establishedby the pre-touch sensors, as well as the perpendicular distance D₂between the second point of application 206 and its respective positionon the second touch interface 205. That second touch input location 209will then be used to generate a three-dimensional segment in accordancewith the method described above.

The previous examples illustrate methods for controlling a virtualobject using a single device. However, situations may arise where morethan one user control the same virtual object. By way of example, andnot by way of limitation, two users may be involved in a virtual world,where they are working together to control a virtual object. FIGS. 3Aand 3B are 3-dimensional schematic diagrams illustrating a method forcontrolling a virtual object in accordance with an embodiment of thepresent invention. FIG. 3A illustrates the method using direct touchtechnology and FIG. 3B illustrates the method using pre-touchtechnology. A first user operates a first device 301, which contains afirst touch interface 303. A second user operates a second device 301′,which contains a second touch interface 305. These two users can beconnected over a network 315. By way of example, and not by way oflimitation, the network may be implemented using wired or wirelessconnection, such as Bluetooth radiofrequency technology, infraredsignals, and the like. The user users may then work in concert tocontrol the same virtual object 313.

The location of the first touch input 307 with respect to the touchinterface 303 of the first device 301 can be determined by the firstuser's finger's F₁ interaction with the first touch interface 303 (i.e.,by direct touch or pre-touch), e.g., as described above. The location ofthe second touch input 309 with respect to the second touch interface305 can be determined by the second user's finger's F₂ interaction withthe second touch interface 305 (i.e., by direct touch or pre-touch) alsopreviously described above. Alternatively, the location of the firstpoint of application 304 and the location of the second point ofapplication 306 may also be determined using pre-touch technology asdescribed in FIG. 2A. The location of the first touch input 307 (orfirst point of application 304) is sent over the network 315 to thesecond device 301′. Similarly, the location of the second touch input309 (or second point of application 306) is sent over the network 315 tothe first device 301.

The location of the first touch input 307 (or first point of application304) can be mapped onto the second device 301′ in accordance with itslocation on or proximity to the first touch interface 303, and willreferred to herein as the location of the projected first touch input307′ (or alternatively the location of the projected first point ofapplication 304′). Similarly, the location of the second touch input 309(or second point of application 306) is mapped onto the first device 301in accordance with its location on or proximity to the second touchinterface 305, and will be referred to herein as the location of theprojected second touch input (or alternatively the location of theprojected second point of application 306′).

The first device 301 can then generate a three-dimensional segment 311using the location of the first touch input 307, the location of theprojected second touch input 309′, and a pre-determined spatialrelationship between the first touch interface and the second touchinterface. Likewise, the second device 301′ will generate athree-dimensional segment 311′ using the location of the second touchinput 309, the location of the projected first touch input 307′ and apre-determined spatial relationship between the first touch interfaceand the second touch interface. By way of non-limiting example, thepre-determined spatial relationship may assume that the first and secondtouch interfaces 303, 305 are in a parallel and overlapping arrangementseparated by a distance d. However, any arbitrary relationship betweenthe two touch interfaces 303, 305 may be used, e.g., one where they arenon-parallel.

Alternatively, the first device 301 may generate a three-dimensionalsegment 311B using the location of the first point of application 304and the location of the projected second point of application 306′, asdescribed in FIG. 2A. Similarly, the second device 301′ may generate athree-dimensional segment 311B′ using the location of the projectedfirst point of application 304′ and the location of the second point ofapplication 306, as described in FIG. 2A. These three-dimensionalsegments 311B, 311B′ may be modified by lengthening or shortening theperpendicular distances D₁, D₂ between the points of application 304,306 and their respective positions 307, 309 on the touch interfaces 303,305. As discussed above, the three-dimensional segment 311B, 311B′generated using the points of application 304, 306 may be preferred overthe three-dimensional segment 311, 311′ generated using the touch inputsbecause of the additional degree of control it provides.

For purposes of our example, assume that the virtual object 313, 313′ ineach device appears to be located between the device's touch interface303, 305 and the plane of the location of the projected touch inputs309′, 307′. Each three-dimensional segment 311, 311′, 311B, 311B′initially passes through the virtual object 313, 313′, and any shift inposition of the three-dimensional segment 311, 311′, 311B, 311B′ willmanipulate the orientation of the virtual object 313, 313′. Because thevirtual object 313, 313′ is being controlled by two separate devices,301, 301′, each device may only control one end of the three-dimensionalsegment 311, 311′, 311B, 311B′. For example, the user of the firstdevice 301 may control the location of the first touch input 307 (orfirst point of application 304), but is unable to control the locationof the projected second touch input 309′ (or projected second point ofapplication 306′) as that is controlled by the user of the second device301′. Likewise, the user of the second device 301′ may control thelocation of the second touch input 309 (or second point of application306), but is unable to control the location of the projected first touchinput 307′ (or projected first point of application 304′) as that iscontrolled by the user of the first device 301. Although both users seethe virtual object 313, 313′ from their own individual perspectives, thecontrol of the virtual object is dependent the interaction between thetwo user's over the network 315.

The control of a virtual object using the methods described above may beexercised in a number of different ways. FIG. 4A-4D are schematicdiagrams illustrating a few examples of possible ways to control avirtual object in accordance with an embodiment of the presentinvention. The location of the first touch input with respect to thefirst touch interface (or location of the first point of application)407 is determined by a first finger's F₁ interaction with the firsttouch interface (i.e., direct touch or pre-touch). Similarly, thelocation of the second touch input with respect to the second touchinterface (or location of the second point of application) 409 isdetermined by a second finger's F₂ interaction with the second touchinterface (i.e., direct touch or pre-touch). For purposes of ourexample, it can be assumed that the virtual object 413 appears to belocated between the first touch interface and the second touchinterface. The location of the first touch input (or location of thefirst point of application) 407, the location of the second touch input(or location of the second point of application) 409, and apre-determined spatial relationship between the two touch interfaces(e.g., actual physical distance between touch interfaces) are used togenerate a 3-dimensional segment 411. The three-dimensional segment 411is then used as a control input for controlling the virtual object 413.The three-dimensional motion segment 411 will initially pass through thevirtual object 413, and any shift in position of the three-dimensionalmotion segment 411 will manipulate the orientation of the virtual object413.

FIG. 4A illustrates a method of rotating the virtual object inaccordance with an embodiment of the present invention. Moving the firstfinger F₁ and the second finger F₂ in opposite directions will cause thevirtual object 413 to rotate. Changes in the orientation of the segment411 can be mapped to changes in the orientation of the object 413. Byadjusting the rate of finger movement, a user can control the rotationalvelocity of the virtual object 413. For example, when the fingers areslowly moved in opposite directions, the virtual object 413 cancorrespondingly rotate slowly. When the fingers are quickly pulled inopposite directions, the virtual object 413 can correspondingly rotatequickly as the orientation of the segment 411 changes. Additionally, onecan control the direction in which the virtual object 413 rotates bychanging the direction in which the fingers are moved. It is importantto note that the virtual object 413 may also be rotated by simplyshifting one finger and leaving the other finger in place. By way ofexample, and not by way of limitation, this motion could propagateindefinitely by shifting one finger a set distance, bringing it back toits original position, and shifting it again a set distance, repeatingthese movements for as long as a user intends to keep the virtual object413 rotating. Furthermore, increasing the length of the threedimensional segment 411 past a certain threshold may indicate a releaseof the rotation.

FIG. 4B illustrates a method for shifting the virtual object inaccordance with an embodiment of the present invention. Moving the firstfinger F₁ and the second finger F₂ in the same direction causes thevirtual object 413 to shift as the segment 411 shifts. If the fingersmove at the same rate, then the virtual object 413 will proceed in astraight line in the direction of the finger movement. To alter theangle at which the virtual object 413 moves, one can adjust the rate ofmovement of one finger relative to the other finger. For example, if thefirst finger F₁ moves at a quicker rate than the second finger, thevirtual object 413 may move in the direction of both fingers, with themovement angled towards the position of the first finger F₁. Similarly,if the second finger F₂ moves at a quicker rate than the first finger,the virtual object 413 may move in the direction of both fingers, withthe movement angled towards the position of the second finger F₂.Additionally, the angle of the three dimensional segment 411 withrespect to the touch interfaces may provide another degree ofdirectional control of the virtual object 413. For example, if thevirtual object 413 takes on the form of a bullet, the bullet will movein a direction perpendicular to the three-dimensional segment 411,dependent on the movement of the fingers.

FIG. 4C illustrates a method for squeezing/compressing a virtual objectin accordance with an embodiment of the present invention. Within thecontext of direct touch, increasing the pressure that the first fingerF₁ exerts on the first touch interface and increasing the pressure thatthe second finger F₂ exerts on the second touch interface will cause thevirtual object 413 to compress. One can imagine exerting differentcombinations of finger pressure on the respective touch interfaces tomodify the compression of the virtual object 413. The process forcompressing a virtual object 413 using pre-touch sensors is slightlydifferent than the process for compressing a virtual object 413 usingdirect touch. Rather than exerting more force on a touch interface,decreasing the perpendicular distance between a finger's point ofapplication and its respective position on the touch interface willcause the virtual object to compress. One can imagine using differentcombinations of perpendicular distances to modify the compression of thevirtual object 413.

FIG. 4D illustrates a method for expanding a virtual object inaccordance with an embodiment of the present invention. Within thecontext of direct touch, decreasing the pressure that the first fingerF₁ exerts on the first touch interface and decreasing the pressure thatthe second finger F₂ exerts on the second touch interface will cause thevirtual object 413 to expand. One can imagine exerting differentcombinations of finger pressure on the respective touch interfaces tomodify the expansion of the virtual object 413. The process forexpanding a virtual object 413 using pre-touch sensors is slightlydifferent than the process for expanding a virtual object 413 usingdirect touch. Rather than exerting less force on a touch interface,increasing the perpendicular distance between a finger's point ofapplication and its respective position on the touch interface willcause the virtual object to expand. One can imagine using differentcombinations of perpendicular distance to modify the expansion of thevirtual object 413.

It is important to note that these are merely a few examples of methodsfor controlling a virtual object. It is possible to use otheralternative methods as well as combining two or more of theaforementioned methods for controlling a virtual object.

FIG. 5 illustrates a block diagram of a computer apparatus 500 that maybe used to implement control of a virtual object as described above. Theapparatus 500 generally includes a processor module 501 and a memory505. The processor module 501 may include one or more processor cores.As an example of a processing system that uses multiple processormodules, is a Cell processor, examples of which are described in detail,e.g., in Cell Broadband Engine Architecture, which is available onlineathttp://www-306.ibm.com/chips/techlib/techlib.nsf/techdocs/1AEEE1270EA2776387257060006E61BA/$file/CBEA_01_pub.pdf, which is incorporated herein by reference.

The memory 505 may be in the form of an integrated circuit, e.g., RAM,DRAM, ROM, and the like. The memory may also be a main memory that isaccessible by all of the processor modules 501. In some embodiments, theprocessor module 501 may include local memories associated with eachcore. A program 503 may be stored in the main memory 505 in the form ofprocessor readable instructions that can be executed on the processormodules 501. The program 503 may be configured to control a virtualobject using two touch interfaces, e.g. as described above with respectto FIG. 1, FIG. 2, FIG. 3, and FIG. 4. The program 503 may be written inany suitable processor readable language, e.g., C, C++, JAVA, Assembly,MATLAB, FORTRAN, and a number of other languages. Input data 507 may bestored in the memory 505. During execution of the program 503, portionsof program code and/or data 507 may be loaded into the memory 505 or thelocal stores of processor cores for parallel processing by multipleprocessor cores. By way of example, and not by way of limitation, theinput data 507 may include locations of touch inputs (both direct touchand pre-touch), and also three-dimensional segments generated by thetouch inputs.

The program 503 may include instructions that when executed by theprocessor 501 implement a method for controlling a virtual object by a)determining a location for a first touch input on a first touchinterface; b) determining a location for a second touch input on asecond touch interface; c) generating a three-dimensional segment usingthe first touch input, the second touch input, and a pre-determinedspatial relationship between the first touch interface and the secondtouch interface; d) manipulating the virtual object using thethree-dimensional segment as a control input; and e) displaying themanipulated virtual object.

The apparatus 500 may also include well-known support functions 509,such as input/output (I/O) elements 511, power supplies (P/S) 513, aclock (CLK) 515, and a cache 517. The apparatus 500 may optionallyinclude a mass storage device 519 such as a disk drive, CD-ROM drive,tape drive, or the like to store programs and/or data. The apparatus 500may also optionally include a display unit 521 and a user interface unit525 to facilitate interaction between the apparatus 500 and a user. Thedisplay unit 521 may be in the form of a cathode ray tube (CRT) or flatpanel screen that displays text, numerals, graphical symbols or images.The user interface 525 may include a keyboard, mouse, joystick, lightpen, or other device that may be used in conjunction with a graphicaluser interface (GUI). The apparatus 500 may also include a networkinterface 523 to enable the device to communicate with other devicesover a network, such as the internet.

The system 500 may optionally include one or more audio speakers 527that are coupled to the processor 501 via the I/O elements 511. Thespeakers can play sounds generated in response to signals generated byexecution of the program 503. The audio speakers 527 can be used, e.g.,when the movement of the virtual object creates a 3-D sound effect. Insome embodiments, the system 500 may include an optional microphone 529,which may be a single microphone or a microphone array. The microphone529 can be coupled to the processor 501 via the I/O elements 511. By wayof example, and not by way of limitation, different 3-D sound effectsmay be generated by tracking both the location of the sound source usingthe microphone array 529 and the location of the virtual object.

Furthermore, by combining tracking a three-dimensional real sound sourcelocation with a microphone array with 3-D virtual object location, onecan generate different 3D sound effect. For example, a virtual mirrorcan be used to bounce real sound in a specific direction so that onlyone of the audio speakers 527 produces corresponding sounds.

The components of the system 500, including the processor 501, memory505, support functions 509, mass storage device 519, user interface 525,network interface 523, and display 521 may be operably connected to eachother via one or more data buses 531. These components may beimplemented in hardware, software or firmware or some combination of twoor more of these.

According to another embodiment, instructions for controlling a virtualobject may be stored in a computer readable storage medium. By way ofexample, and not by way of limitation, FIG. 6 illustrates an example ofa non-transitory computer-readable storage medium 600 in accordance withan embodiment of the present invention. The storage medium 600 containscomputer-readable instructions stored in a format that can be retrieved,interpreted, and executed by a computer processing device. By way ofexample, and not by way of limitation, the computer-readable storagemedium 600 may be a computer-readable memory, such as random accessmemory (RAM) or read only memory (ROM), a computer readable storage diskfor a fixed disk drive (e.g., a hard disk drive), or a removable diskdrive. In addition, the computer-readable storage medium 600 may be aflash memory device, a computer-readable tape, a CD-ROM, a DVD-ROM, aBlu-Ray, HD-DVD, UMD, or other optical storage medium.

The storage medium 600 contains instructions 601 configured to controland manipulate a virtual object through touch. The Instructions forControlling a Virtual Object 601 may be configured to implement controlof a virtual object in accordance with the methods described above withrespect to FIG. 1, FIG. 2, FIG. 3, and FIG. 4. In particular, theInstructions for Controlling a Virtual Object 601 may includeDetermining Location of First Touch Input on First Touch InterfaceInstructions 603 that are used to determine the location of a firsttouch input with respect to the first touch interface. The Instructionsfor Controlling a Virtual Object 601 may further include DeterminingLocation of Second Touch Input on Second Touch Interface Instructions605 that are used to determine the location of a second touch input withrespect to the second touch interface. The locations of the touch inputsmay be determined by way of direct contact or pre-touch sensors asdescribed above with respect to FIGS. 1A-1C and FIGS. 2A-2C. Theseinstructions may also be configured to save each user's individualizedtouch arrangement to take into account individual finger lengths andfinger movement rates.

The Instructions for Controlling a Virtual Object 601 may also includeGenerating a Three Dimensional Segment Instructions 607 that are used togenerate a three-dimensional segment using the location of the firsttouch input, the location of the second touch input, and apre-determined spatial relationship between the first touch interfaceand a second touch interface. This may trigger one or more ManipulatingVirtual Object Instructions 609 that manipulate the orientation of thevirtual object using the three-dimensional segment as a control input.

The Instructions for Controlling a Virtual Object 601 may furtherinclude Displaying Manipulated Virtual Object Instructions 611 that areused to display the manipulated virtual object to one or more users.These instructions may also include directions for playing particularsound effects (e.g., 3-D sound effects) in accordance with anythree-dimensional control segment generated or any finger movementsmade.

Although the present invention has been described in considerable detailwith reference to certain preferred versions thereof, other versions arepossible. For example, although certain embodiments are described inwhich the three-dimensional segment is defined in terms of Cartesiancoordinates, those of skill in the art will recognize that the segmentcan be defined in terms of other coordinate systems, such as cylindricalor polar spherical coordinates. In addition it is noted that someembodiments are described in which an overlapping parallel relationshipis assumed between the first and second touch interfaces. However,embodiments of the invention may be implemented with any relationshipbetween the two touch interfaces. For example, the two interfaces may bein a non-parallel, e.g., mutually perpendicular, relationship.Furthermore, there may be a skew or lateral offset between the first andsecond touch interfaces. Therefore, the spirit and scope of the appendedclaims should not be limited to the description of the preferredversions contained herein. Instead, the scope of the invention should bedetermined with reference to the appended claims, along with their fullscope of equivalents.

All the features disclosed in this specification (including anyaccompanying claims, abstract and drawings) may be replaced byalternative features serving the same, equivalent or similar purpose,unless expressly stated otherwise. Thus, unless expressly statedotherwise, each feature disclosed is one example only of a genericseries of equivalent or similar features. Any feature, whether preferredor not, may be combined with any other feature, whether preferred ornot. In the claims that follow, the indefinite article “A”, or “An”refers to a quantity of one or more of the item following the article,except where expressly stated otherwise. Any element in a claim thatdoes not explicitly state “means for” performing a specified function,is not to be interpreted as a “means” or “step” clause as specified in35 USC §112, ¶6. In particular, the use of “step of” in the claimsherein is not intended to invoke the provisions of 35 USC §112, ¶6.

The reader's attention is directed to all papers and documents which arefiled concurrently with this specification and which are open to publicinspection with this specification, and the contents of any papers anddocuments incorporated herein by reference.

What is claimed is:
 1. A method for controlling a virtual object,comprising: determining a location for a first touch input on a firsttouch interface; determining a location for a second touch input on asecond touch interface; generating a three-dimensional segment using thelocation of the first touch input, the location of the second touchinput, and a pre-determined spatial relationship between the first touchinterface and the second touch interface; and manipulating the virtualobject using the three-dimensional segment as a control input, whereinmanipulating the virtual object using the three-dimensional segment as acontrol input includes generating a new three-dimensional segment inresponse to a shift in the location for the first input touch and/or thelocation for the second input touch.
 2. The method of claim 1, whereinthe first touch interface is a touch screen.
 3. The method of claim 2,wherein the second touch interface is a touch pad.
 4. The method ofclaim 1, wherein the first or second touch interface includes a curvedsurface.
 5. The method of claim 1, further comprising displaying themanipulated virtual object on a display.
 6. The method of claim 1,wherein the second touch interface is on a different device than thefirst touch interface.
 7. The method of claim 1, wherein the secondtouch interface is the same device as the first touch interface.
 8. Themethod of claim 1, further comprising displaying the manipulated virtualobject on a display, wherein displaying the manipulated virtual objecton a display includes displaying the virtual object on a displayseparate from a device containing the first touch interface and a devicecontaining the second touch interface.
 9. The method of claim 1, furthercomprising displaying the manipulated virtual object on a display,wherein displaying the manipulated virtual object on a display includesdisplaying the virtual object on a first touch screen.
 10. The method ofclaim 9, wherein displaying the manipulated virtual object on a displayalso includes displaying the virtual object on a second touch screen.11. An apparatus for controlling a virtual object, comprising: a firsttouch interface; a second touch interface; a processor operably coupledto the first touch interface; and instructions executable by theprocessor configured to: a) determine a location of a first touch inputon the first touch interface; b) determine a location of a second touchinput on the second touch interface; c) generate a three-dimensionalsegment using the location of the first touch input, the location of thesecond touch input, and a pre-determined spatial relationship betweenthe first touch interface and the second touch interface; d) manipulatethe virtual object using the three-dimensional segment in c) as acontrol input, wherein manipulating the virtual object using thethree-dimensional segment as a control input includes generating a newthree-dimensional segment in response to a shift in the location for thefirst input touch and/or the location for the second input touch; and e)display the manipulated virtual object.
 12. The apparatus in claim 11,wherein the processor is further operably coupled to the second touchinterface; and the first touch interface and the second touch interfaceare both situated on a case having first and second major surfaces. 13.The apparatus in claim 12, wherein the first touch interface is a touchscreen located on the first major surface.
 14. The apparatus in claim13, wherein the second touch interface is a touch pad located on thesecond major surface.
 15. The apparatus in claim 11, wherein the firsttouch interface in a) is situated on a first device and the second touchinterface in b) is situated on a second device.
 16. The apparatus inclaim 15, wherein the first device and the second device are connectedover a wireless network.
 17. The Apparatus of claim 11, wherein thefirst or second touch interface includes a curved surface.
 18. Theapparatus in claim 11, further comprising a visual display separate froma device containing the first touch interface and a device containingthe second touch interface.
 19. A computer program product comprising: anon-transitory computer-readable storage medium having computer readableprogram code embodied in said medium for controlling a virtual objectusing two touch interfaces, said computer program product having:computer readable program code means for determining a location of afirst touch input on a first touch interface; computer readable programcode means for determining a location of a second touch input on asecond touch interface; computer readable program code means forgenerating a three-dimensional segment using the location of the firsttouch input, the location of the second touch input, and apre-determined spatial relationship between the first touch interfaceand the second touch interface; and computer readable program code meansfor manipulating the virtual object using the three-dimensional segmentin as a control input, wherein manipulating the virtual object using thethree-dimensional segment as a control input includes generating a newthree-dimensional segment in response to a shift in the location for thefirst input touch and/or the location for the second input touch.